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FOREWORD 


Under  the  Sui*veillanee  Systems  ROT  E  Project,  the  COMPOIIEIIT  IOTEGRA- 
TION  Task  seeks  to  evaluate  total  system  configurations  of  men,  machines, 
and  procedures,  using  an  experimental  computer-bused  interpretation 
facility. 

The  present  feasibility  study  was  conducted  by  Wayne  H.  Jone 3  of  the 
Advanced  Systems  Division,  System  Development  Corporation,  under  contract 
to  the  Department  of  the  Army.  Research  Memorandum  66-1  is  based  on  a 
search  of  the  literature  for  reports  of  similar  simulations  and  for 
characteristics  of  systems  which  have  bccn--or  could  be--amenable  to 
simulation.  Concurrently,  system  analyses  of  image  interpretation 
systems  were  conducted  for  the  p- appose  of  formulating  a  suitable  frame¬ 
work  for-  simulation.  In  particular,  models  were  sought  which  contained 
man-maehinc  interactions  as  parameters. 


FEASIBILITY  OF  COMPUTER  SIMULATION  OF  AN  IMAGERY  INTERPRETATION  SYSTEM 


A  means  of  quiclcly  and.  reliably  evaluating  alternative  tactical 
image  interpretation  facilities  is  needed.  The  objective  of  the  present 
study  was  to  examine  the  feasibility  of  simulating  image  interpretation 
systems  on  a  computer  in  order  to  aid  in  the  selection  of  promising  man- 
machine  configurations  for  further  laboratory  and  field  evaluation. 

I 

Research  in  image  systems  is  complicated  by  the  many  factors  that 
can  influence  system  performance.  Input  load,  system  personnel,  opera¬ 
tions,  equipment,  environment,  and  required  system  outputs  can  critical¬ 
ly  affect  system  performance.  Each  of  these  factors  can,  in  turn,  be 
divided  into  many  component  factors  or  variables  which  separately  or  in 
combination  affect  performance.  Input  load,  for  example,  can  be  express¬ 
ed  in  terms  of  batches  of  imagery  varying  in  type,  length,  number  of 
frames,  scale,  quality,  target  density  and  distribution,  terrain  type, 
etc.  Even  more  factors  or  variables  must  be  considered  in  analyzing 
system  operations,  equipment,  and  environment.  The  difficulty  of 
controlling  or  systematically  varying  any  large  number  of  these  variables 
even  within  a  well -designed  experimental  facility  has  acted  as  an  inhib¬ 
itory  influence  on  the  scope  and  direction  of  experimental  research 
conducted  in  image  systems. 

The  aim  of  the  present  study  was  to  investigate  the  feasibility 
of  using  a  computer  to  simulate  the  factors  influencing  system  per¬ 
formance.  The  goal  of  such  simulation  is  to  develop  tentative  configu¬ 
rations  for  the  effective  use  of  personnel  and  equipment  within  image 
systems,  configurations  which  can  later  be  tested  experimentally.  If 
appropriate  representation  of  system  imagery  inputs,  reporting  require¬ 
ments,  interpreters,  equipment,  processing  procedures,  and  intelligence 
products  can  be  achieved  in  a  model,  then  the  manipulation  of  the  many 
parameters  involved  could  probably  be  achieved  more  easily  using  a  com¬ 
puter  than  in  the  laboratory.  Only  those  equipment -personnel  configu¬ 
rations  and  processing  which  showed  the  most  promise  of  meeting  the 
system's  input  load  and  output  objectives  would  then  be  tried  out  in 
the  Surveillance  Research  Laboratory  of  the  U.  S.  Army  Personnel  Re¬ 
search  Office  (U.  S.  APRO)  or  in  field  system  tests  and  operations. 

For  the  simulation  to  be  effective,  it  is  necessary  to  construct 
a  valid  representation  of  the  probability  of  target  detection  and  iden¬ 
tification  in  terms  of  characteristics  of  imagery,  interpreters,  equip¬ 
ment,  and  procedures.  Deterministic  and  stochastic  models  and  the 
simulation  of  system  performance  using  either  analytical  or  Monte  Carlo 
methods  were  considered  in  this  feasibility  study. 


GENERAL  MODEL  FOR  SYSTEMS  ANALYSIS 


The  effectiveness  of  a  system  can  be  represented  by  the  equation 

E  »  f(xt,  yj) 

where  E  is  the  effectiveness,  x^the  variables  subject  to  control,  and  y^ 

the  variables  not  subject  to  control  (l ) •  Restrictions  on  the  variables 
my  be  expressed  in  supplementary  equations  and  /or  inequalities.  In 
these  terms,  a  problem  of  interest  is  to  determine  the  values  x^  which 

maximize  E  for  various  given  sets  of  values  of  yy  If  the  function  f 

can  be  expressed  mthematically,  the  maximum  value  of  E  can  sometimes 
be  found  by  mathematical  analysis.  If  direct  analysis  fails,  there 
are  mny  schemes  which,  through  an  iterative  process,  yield  an  appreci¬ 
ation  to  the  maximum.  Systems  solvable  by  these  means  Include  those  in 
which  inventory,  allocation,  waiting  time,  replacement,  and  competitive 
processes  are  involved.  Of  these,  competitive  models  are  perhaps  the 
most  difficult  because  the  solution  depends  on  assumptions  about  the 
behavior  of  opposing  elements,  quite  often  some  of  the  variables  are 
random,  in  which  case  the  objective  is  usually  the  maximization  of  the 
expected  value  of  E.  For  many  large  systems,  however,  no  tractable  mathe¬ 
matical  model  is  available.  For  some  of  these  systems,  models  of  system 
components  are  known,  but  the  total  systems  are  too  complex  for  mathema¬ 
tical  analysis.  Computer  simulation  is  often  valuable  in  these  cases. 

If,  as  is  likely,  random  variables  are  present,  then  estimates  of  means 
and  variances  of  system  effectiveness  may  be  obtained  for  various  values 
of  the  control  variables  x±.  These  may  be  used  as  a  basis  for  selection 

of  a  set  of  promising  system  configurations  for  futher  investiga¬ 

tion.  If  the  cooputer  model  is  a  sufficiently  precise  representation  of 
the  real  system,  the  last  step  is  unnecessary;  unfortunately,  the  pre¬ 
cision  of  the  model  is  seldom  known. 

Most  large  systems  are  further  complicated  by  multiple  criteria,  or 
objectives.  Usually,  one  criterion  is  maximized  at  the  expense  of  the 
others,  and  the  best  choice  from  several  configurations  cannot  be  deter¬ 
mined  without  recourse  to  Judgment  factors  external  to  the  model. 


IMAGE  SYSTEM  APPLICATION 

For  luagery  interpretation,  system  effectiveness  measures  are  usually 
taken  to  be  accuracy,  completeness,  and  timeliness.  Control  variables  x^ 
are  represented  by  sensors,  platforms,  quality  and  quantity  of  imagery, 
interpreter  aptitude  and  state  of  training,  and  procedures.  Variables  y^ 
not  subject  to  control  are  represented  by  target  distribution,  weather, 
terrain,  and  mission. 
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The  feasibility  of  ~>mi  ter-  Simula*  ion  of  a  system  depends  heavily 
on  the  goals.  To  the  extend  tlixt  then:  goals  include  determination  of 
the  effect  on  system  performance  of  changing  certain  variables,  the 
system  model  must  provide  for  these  variables,  directly  or  indirectly. 
For  example,  a  particular  sensor  need  not  be  represented  explicitly  by 
a  single  control  variable.  The  representation  might  be  a  point  in  the 
parameter  space  of  image  quality.  In  such  a  model,  nonexistent  sensors 
could  be  tested,  provided,  of  course,  that  the  validity  of  the  model 
extends  to  the  appropriate  region  of  the  parameter  spuce. 


The  spectrum  of  control  variables  of  interest  in  image  systems  is 
illustrated  in  Table  1.  Note  that  most  of  the  variables  arc  discrete 
(usually  two  categories)  rather  tlian  continuous.  Categorical  variables 
arc  inherently  more  difficult  to  model.  In  the  first  place,  separate 
prediction  equations  are  required  for  each  combination  of  categories -- 
unless  the  interactions  are  negligible.  The  usual  analysis  of  variance 
model  for  20  dichotomous  variables  involves  : 'll  parameters,  if  only  the 
main  effects  and  two-factor  interactions  are  included.  In  the  absence 
of  knowledge  of  the  structure  of  these  variables,  the  parameters  must 
be  estimated  from  a  very  large  and  com]  lex  experiment.  But  the  data 
resulting  from  this  experiment  would  already  contain  the  answers  to  the 
questions  which  would  be  asked  of  a  computer  simulation,  namely,  which 
combination  of  variables  produces  the  best  system  performance,  and  how 
does  one  configuration  compare  with  another?  Therefore,  computer  simu¬ 
lation  of  this  kind  of  model  will  not  provide  any  new  information. 


Other  investigators  have  readied  similar  conclusions  concerning 
the  feasibility  of  this  type  of  computer  simulation.  In  on  unpublished 
Project  Michigan  report  (5),  DeVoe  and  Hoagbin  state: 

"A  complex  man-machine  system,  such  as  a  Tactical 
Image  Interpretation  Facility,  docs  not  lend  itself  to 
ordinary  engineering  analysis.  It  is  not  possible  to  write 
functions  relating  performance  to  design  variables  that 
lend  themselves  to  mathematical  operations.  Many  param¬ 
eters,  for  example,  image  quality,  cannot  be  quantified 
satisfactorily . " 


In  a  more  general  context,  the  following  paragraph  is  found  in 
Muckier  and  Cfbermayer  (4 ) : 

"A  number  of  mathematical  models  can  be  fitted  to 
human  operator  data,  but  only  a  quasi-linear  model  has 
been  extensively  tested.  Simulations  are  becoming  in¬ 
creasingly  sophisticated,  but  field  tests  and  empirical 
methods  are  used  because  human  performance  is  unpre¬ 
dictable.  Despite  the  large  amount  of  data  gathered, 
little  understanding  of  it  has  emerged.  Concepts  and 
theory  remain  in  a  state  of  flux." 


Ornstein  and  others  (5,  6)  of  North  American  Aviation  have  developed 
an  elaborate  computer  model  of  search  system  performance  which  deserves 
mention.  In  this  model,  there  is  one  vehicle  or  platform  which  carries 
one  or  more  observer -display  combinations.  Only  one  target  is  assumed, 


IMAGE  SYSTEM  CONTROL  VARIABLES' 


1.  Imagery  Screening  and  Hot  Reporting  (screening  techniques,  multi 
pie  displays  and  display  times) 

2.  Report  Composing  (oral,  typewriter,  or  special  keyboard?) 

3*  CRT  vs.  Projection  Viewing  (photo,  IR,  SIAR) 

4.  Direct  Viewing  vs.  Projection  Viewing  for  Screening 

5.  Comparative  Viewing  (methods  of  presentation  of  imagery) 

6.  Team  Interpretation  Techniques 

7.  Optimal  Viewing  Angles 

8.  Direct  vs.  Projected  Magnification  of  Imagery 

9.  Resolution  Loss  in  Projection 

10.  Manual  vs.  Automated  Reference  Retrieval 

11.  Contribution  of  Keys  and  Other  Reference  Material 

12.  Multisensor  Imagery  -  Comparative  Cover  (value  of  partial  or 
complete  IR,  SIAR  records  in  photo  interpretation) 

13.  Immediate  and  Detailed  Interpretation -Parametric  Data  on  Methods 
of  Search  (instructions  to  II  varied) 

14.  Role  of  Officer-in-Charge  in  Assignments  and  in  Resolving 
Interpreter  Disagreements 

13*  Performance  of  IIs  on  Specialized  Interpretation  Tasks  (inter¬ 
changeability  of  IIs) 

16.  Work-Rest  Cycles 

17.  Positive  vs.  Negative  Transparencies 

18.  Effect  of  Background  Noise 

19-  Forced  vs.  Self -Pacing  and  Viewing 

20.  Discrete  vs.  Continuous  Movement  of  Imagery 

21.  Manual  vs.  Automatic  Plotting 

22.  Effect  on  Target  Location  of  Inaccuracy  of  Platform  Location 

23.  Vertical,  Oblique,  Panoramic  Imagery 

24.  Manual  vs.  Computer  Mensuration 

25.  Manual  Methods  of  Measuring  Height 

26.  Cursor  Positioning  Accuracy 


Abstracted  from  Applied  Psychology  Corporation  Interim  Report  (2) 


and  the  mission  ends  when  tv/  target  has  been  identified  (correctly  or 
not.)  or  .'hen  the  target  area  h.  .  been  completely  searched  without  a  posi¬ 
tive  identification.  The  si  arch  is  performed  in  reul  time  from  dynamic 
i'. -plays,  and  the  search  pa  inu.\  oe  modified  to  re-examine  a  ground 
patch  if  any  observer  repo  to  detection  but  not  positive  identification. 
The  model  is  essentially  a  Markov  chain  over  discrete  time  intervals 
and  the  target  -rea  segments,  with  13  states  defined  by  various  response 
conditions.  The  necessary  transition  pi'obabilities  arc  determined 
empirically. 

There  are  obvious  differences  between  this  type  of  search  and  tacti¬ 
cal  imagery  interpretation  within  an  interpretation  facility.  In  partic¬ 
ular,  the  assumption  of  a  single  target  in  a  mission  makes  this  process 
quite  different  from  the  one  usually  studied  experimentally  within  the 
Surveillance  Research  Laboratory.  Of  greater  significance,  however,  is 
the  dependence  of  system  performance  on  the  transition  probabilities. 
These  probabilities  are  clearly  functions  of  observers,  sensors,  plat¬ 
forms,  terrain,  weather,  and  many  other  factors.  In  order  to  determine 
the  effect  on  system  performance  of  varying  any  of  these  factors,  it  is 
necessary  to  provide  the  model  with  probability  parameters  appropriate 
to  each  system  configuration  to  be  studied.  Within  the  present  state 
of  knowledge,  it  does  not  appear  that  these  detailed  aspects  of  the 
search  process  are  any  better  known  than  total  system  performance. 

Another  type  of  simulation  which  should  be  mentioned  calls  for  human 
beings  to  play  an  active  part  in  the  simulation.  Such  simulations  have 
been  conducted  in  the  Logistics  Systems  Laboratory  of  the  Rand  Corpora¬ 
tion  in  order  to  assist  the  Air  Force  in  the  evaluation  of  logistics 
policies.  Simulations  were  conducted  using  experienced  logistics 
officers,  with  parts  replacement  needs  computed  from  failure  rate  distri¬ 
butions.  Murray  Geisler  (7)  reports  that  each  of  three  simulation  ex¬ 
periments  cost  more  than  $1,000,000,  lasted  about  two  years,  and  involved 
more  than  100  people.  Further  details  on  the  programming  task  are  avail¬ 
able  in  Little  and  Shelton  (8).  These  experiments  are  illustrative  of 
the  considerable  effort  required  to  simulate  a  complex  system,  even 
when  man,  the  most  complex  system  component,  is  not  simulated. 


DISCUSSION  AND  CONCLUSIONS 


Faced  with  the  problem  of  system  design,  which  includes  choices  of 
hardware  (including  some  not  yet  developed),  allocation  of  functions  among 
men  and  machines,  and  specification  of  procedures,  designers  may  find  that 
it  is  too  expensive  in  both  time  and  money  to  build  even  prototypes  of 
versions  to  be  considered.  On  the  other  hand,  the  large  number  of  inter¬ 
actions  present  in  large  scale  systems  means  that  even  the  most  carefully 
designed  set  of  flow  charts  and  specifications  may  not  result  in  an  oper¬ 
ating  system  which  satisfies  the  requirements.  An  alternative  course  of 
action  is  to  construct  a  simulation  model  which  can  be  programmed  and 
run  on  a  computer.  The  model  may  be  partially  validated  on  the  current 
operational  system,  but  the  ultimate  value  of  the  simulation  lies  in  its 
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uoo  ia  on  .uiv.  .it>  a  reg.  .  tin  i.v.r:  Jnctcr  space,  luunt-ly,  tiiat  which 
contain:-  the  possible  systems  to  be  considered  but  not  yet  built ^ 

It  scene  apparent  tliat  July  system  configurations  whose  characteris¬ 
tics  arc  representable  within  t!;>  model  parameter  space  can  be  evaluated 
by  uoiivj  the  model.  Put  another  x.y,  the  effect  of  a  particular  variable 
on  system  performance  caiuiot  be  determined  unless  that  variable  is  present 
in  the  model,  explicitly  or  implicitly. 

As  of  now,  the  surveillance  system  performance  measures  of  accuracy 
and  completeness  cannot  be  expressed  as  functions  of  the  <•'  >'  .ri- 

ables  appearing  in  Table  1,  which  arc  typical  of  problems  being  studied 
in  U.  S.  APRO.  The  reason  seems  to  be  tliat  the  human  function  of  imago 
interprets tion  is  complex  and  little  understood,  and  cannot  be  broken 
dovn  into  components  whose  operation  is  well  understood.  This  state  of 
affairs  is  in  sharp  contrast  to  subsystems  consisting  of  hardware  only, 
where  total  performance  can  be  predicted  because  of  detailed  knowledge 
of  component  operation.  It  is  therefore  the  conclusion  of  this  study 
tliat  computer  simulation  of  an  image  interpretation  system  is  not  currently 
feasible. 

The  feasibility  of  simulation  should  be  explored  again  in  the  future 
when  the  data  necessary  for  the  successful  conduct  of  a  computer  simula¬ 
tion  ‘nave  been  collected. 

The  conclusion  applies  only  to  accuracy  and  completeness;  recent 
work  by  o.  S.  APRO  and  Nortronics  suggests  the  usefulness  of  a  computer 
model  of  the  system  as  a  network  of  queues  of  subtasks,  where  estimates 
of  subtask  performance  times  ore  provided  initially  by  expert  judgment 
and  refined  later  through  observation  in  the  laboratory.  It  will  be 
assumed  in  this  model  tliat  the  times  are  for  an  average  level  of  perform¬ 
ance.  This  assumption  is  troublesome,  but  it  may  be  removed  eventually 
or  alleviated  by  gradually  incorporating  accuracy  and  completeness 
criteria  as  knowledge  of  these  variables  increases  with  laboratory  ex¬ 
perimentation.  At  any  rate,  there  Is  a  large  body  of  experience  to  sup¬ 
port  the  use  of  exponential  distributions  of  service  time  to  describe 
times  of  execution  of  all  sorts  of  tasks.  A  characteristic  property  of 
this  family  of  distributions  is  tliat  the  number  of  tasks  completed  in 
a  fixed  time  interval  has  the  Poisson  distribution.  The  usefulness  of 
such  a  model  does  not  depend  on  this  or  any  other  specific  distribution, 
however.  At  the  very  least,  it  would  perform  time  line  analyses  on  the 
computer  which  are  laborious  said  time  consuming  when  done  by  hand. 
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